Expansion of high performing placement criteria

ABSTRACT

Systems and methods of evaluating information in a computer network environment are provided. A data processing system can obtain or receive a content placement criterion, such as a keyword, associated with a content item and can determine a quality metric of the content placement criterion. The data processing system can identify a candidate content placement criterion and expand placement criteria associated with the content item to include the content placement criterion and the candidate content placement criterion based at least in part on an evaluation of the quality metric of the content placement criterion. The data processing system can expand placement criteria based in part on a throttling parameter. The data processing system can identify a correlation between a document and the placement criteria to identify appropriate content items for the document.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation application of and claims priority toU.S. application Ser. No. 16/388,953, filed on Apr. 19, 2019, which is acontinuation of U.S. application Ser. No. 15/432,083 (now U.S. Pat. No.10,311,472), filed on Feb. 14, 2017, which is a continuation of U.S.application Ser. No. 15/066,225 (now U.S. Pat. No. 9,607,314), filed onMar. 10, 2016 which is a continuation of U.S. application Ser. No.13/532,335 (now U.S. Pat. No. 9,317,487), filed on Jun. 25, 2012, theentire contents of which are hereby incorporated by reference.

BACKGROUND

In a networked environment such as the Internet, it is often desirableto pair one or more documents with a requested document, such as a webpage. A document viewer's experience may be enhanced by content that isof interest to the viewer. For example, a requested web page may bedisplayed along with one or more additional content items. The contentcan be selected based on a variety of criteria, often with the end goalof soliciting a user reaction such as exploration of the content via aclick-through.

SUMMARY

Aspects and implementations of the present disclosure are directed tosystems and methods of evaluating information in a computer networkenvironment. A data processing system can obtain or receive a contentplacement criterion, such as a keyword, associated with content itemsand can determine a quality metric of the content placement criterion.The data processing system can identify a candidate content placementcriterion and expand placement criteria associated with the content itemto include the content placement criterion and the candidate contentplacement criterion based at least in part on an evaluation of thequality metric of the content placement criterion. The data processingsystem can expand placement criteria based in part on a throttlingparameter. The data processing system can identify a correlation betweena document and the placement criteria to identify one or moreappropriate content items for the document. Content items may includetext, images, and media elements such as an image, video, or interactivefeature (e.g., a game).

A content provider may provide content items to be delivered withdocuments from content publishers. In some implementations, theplacement system determines what content items to deliver based onplacement criteria. In some implementations, successful placementcriteria used to place a content item are expanded to include additionalplacement criteria derived from a successful placement criterion.

At least one aspect is directed to a computer-implemented method ofevaluating information in a computer network environment by obtaining acontent placement criterion associated with a content item; determininga quality metric of the content placement criterion; identifying acandidate content placement criterion; and expanding placement criteriaassociated with the content item to include the content placementcriterion and the candidate content placement criterion based at leastin part on an evaluation of the quality metric of the content placementcriterion. In some implementations, where the quality metric is based ona click through rate, the method further includes determining the clickthrough rate of the content placement criterion corresponding to thecontent item.

At least one aspect is directed to a system of providing information viaa computer network, the system including a data processing system havingat least one of a content placement module and a quality metric module,the data processing system configured to receive a content placementcriterion associated with a content item; determine a quality metric ofthe content placement criterion; identify a candidate content placementcriterion; and expand the placement criteria associated with the contentitem to include the content placement criterion and the candidatecontent placement criterion based at least in part on the quality metricof the content placement criterion.

At least one aspect is directed to a computer readable storage mediumhaving instructions to provide information on a computer network, theinstructions including instructions to receive a content placementcriterion associated with a content item; determine a quality metric ofthe content placement criterion; identify a candidate content placementcriterion; and expand the placement criteria associated with the contentitem to include the content placement criterion and the candidatecontent placement criterion based at least in part on an evaluation ofthe quality metric of the content placement criterion. In someimplementations the instructions include instructions to determine athrottling parameter of the content item and expand the contentplacement criterion to include the candidate content placement criterionbased on an evaluation of the throttling parameter.

These and other aspects and implementations are discussed in detailbelow. The foregoing information and the following detailed descriptioninclude illustrative examples of various aspects and implementations,and provide an overview or framework for understanding the nature andcharacter of the claimed aspects and implementations. The drawingsprovide illustration and a further understanding of the various aspectsand implementations, and are incorporated in and constitute a part ofthis specification.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings are not intended to be drawn to scale. Likereference numbers and designations in the various drawings indicate likeelements. For purposes of clarity, not every component may be labeled inevery drawing. In the drawings:

FIG. 1 is a block diagram depicting an example environment to evaluateinformation on a computer network, according to an illustrativeimplementation;

FIG. 2 is a block diagram illustrating a general architecture for acomputer system that may be employed to implement various elements ofthe systems and methods described and illustrated herein, according toan illustrative implementation;

FIG. 3 is a block diagram depicting relationships between information,according to an illustrative implementation; and

FIG. 4 is a flow diagram depicting a method of evaluating information ina computer network.

DETAILED DESCRIPTION

Following below are more detailed descriptions of various conceptsrelated to, and implementations of, methods of, apparatuses for, andsystems for expanding keywords to include high performing keywords. Thevarious concepts introduced above and discussed in greater detail belowmay be implemented in any of numerous ways, as the described conceptsare not limited to any particular manner of implementation. Examples ofspecific implementations and applications are provided primarily forillustrative purposes.

A content provider such as a content sponsor may provide content itemsto be delivered with documents from primary content publishers. When aweb page or other document is requested, e.g., by a user, a contentplacement system can determine one or more content items to present withthe document. In some implementations, the placement system determineswhat content item or items to deliver based on placement criteria. Forexample, the placement system may identify a correlation between thedocument and a placement criterion associated with a content item andselect that content item for delivery. One example of a placementcriterion is a keyword. A content item may include text, images, andmedia elements such as an image, video, or interactive feature (e.g., agame). Content items may provide information enhancing other contentwithin a document, e.g., primary content. Content items may be relatedto the primary content (e.g., behind-the-scenes media), include materialrelated to the source of the primary content (e.g., highlights of othercontent from the source), include material that may be of interest tothe user based on user-related factors (e.g., a relevant weatherreport), provide offers for the user to participate in a transaction, orprovide media elements such as a video player or interactive game.

In some implementations, a placement criterion is successful and leadsto the placement of a content item with a web page where the viewer ofthe web page is likely to respond to the content item. The placementcriteria can also be expanded, e.g., from criteria originally providedfrom the content provider to the content placement system, to includeadditional placement criteria. The additional placement criteria can bederived from the successful placement criterion. In someimplementations, the content item is part of a content group where theplacement criteria are affiliated with the group as a whole. In thisexample, successful placement criteria for a content item from thecontent group may be expanded to include additional placement criteriafor additional content items or other content in the group.

FIG. 1 illustrates an example system 100 for providing content to a userdevice 126 via a computer network such as network 110 according to animplementation of the present disclosure. The network 110 can includecomputer networks such as the Internet, local, metro, or wide areanetworks, intranets, and other communication networks such as mobiletelephone networks. The network 110 can be used to access web pages thatcan be displayed on at least one user device 126, such as a laptop,desktop, tablet, personal digital assistants, smart phones, or portablecomputers. For example, via the network 110 a user of user device 126can access web pages provided by a content publisher 120. In thisexample, the user device 126 uses software, such as a web browser, toaccess a web server and retrieve a web page for display on a monitor ofthe user device 126. The web server may be the content publisher 120 oran intermediary host.

The system 100 includes at least one data processing system 150 that,for example, can include at least one processor or other logic device tocommunicate via the network 110 with at least one content publisher 120and at least one supplemental content provider 122. In oneimplementation, the data processing system 150 includes a contentplacement system configured to evaluate and include candidate keywordsas placement criteria used to place content items with web pages orother documents via the network 110. The data processing system 150 canalso include at least one content placement module 154, at least onequality metric module 158, and at least one data repository or database170. The data processing system 150 can include or communicate with acontent selection server, a content host server, a content placementserver, and other data processing systems.

In some implementations, the data processing system 150 includes aninterface configured to receive a request via the network 110 toidentify content items for delivery with a web page. The data processingsystem 150 may receive the request in real-time, e.g., after the userdevice 126 requests access to a web page and before the web page isfully rendered on the user device 126. The activity of users on thenetwork 110, e.g., requests for a web page, can be maintained in ananonymous fashion such that individual users cannot be identified fromthe maintained user activity. Furthermore, user activity information canbe collected on an opt-in or opt-out basis. Users can control thecollection of their activity information. The user can be represented byidentifiers associated with the user device 126, for example using acookie, without regard to the actual identity of the individual personaccessing the user device 126. Identifiers may be selected containing nopersonally-identifiable information about the user or the user device126, for example, a random number may be used. In some implementations,a user may affirmatively opt to provide personally identifyinginformation such as a name, nickname, e-mail address, or otheridentifying feature to the data processing system 150. In someimplementations, a user may benefit without providing any suchpersonally identifying information to the data processing system 150.

FIG. 2 is a block diagram of a general architecture of an illustrativecomputer system 200. In some implementations, the data processing system150 of FIG. 1 includes the computer system 200. For example, the contentplacement module 154 or the quality metric module 158 can each be partof at least one computer system 200. The example computer system 200includes one or more processors 250 in communication, via a bus 215,with one or more network interfaces 210 (in communication with thenetwork 110), I/O interfaces 220 (for interacting with the user), andinternal memory 270. In some implementations, a processor 250incorporates, or is directly connected to, additional cache memory 275.In some implementations, additional components are in communication withthe computer system 200 via a peripheral interface 230. In someimplementations, such as a server, there is no I/O interface 220, or theI/O interface 220 is not used. The computer system 200 of FIG. 2 may beemployed to implement the systems and methods discussed herein.

In some implementations, the user device 126 includes a computer system200. For example, a user can interact with an input device 224, e.g., akeyboard, mouse, or touch screen, to request a web page to be deliveredover the network 110, received at interface 210, and output via outputdevice 226, e.g., a display, screen, touch screen, or speakers. Therequest can be processed by the data processing system 150, which mayinclude a computer system 200, for example to identify content itemsfrom content provider 122 for delivery with content from contentpublisher 120 based on placement criteria.

When a web page is requested by a user device 126, a content placementsystem, e.g., data processing system 150, determines one or more contentitems to present with the requested web page. In some implementations,the placement system determines what content items to deliver based onplacement criteria.

A placement criterion, such as a keyword, can include more than oneterm, component, number, or word. The placement criteria can include atleast one keyword used by the data processing system to match thecontent item with a web page (or other document). One placementcriterion can include a phrase having more than one word, e.g.,“BrandName Product.” In an example usage, the data processing system 150receives notification of a request for access to a web page. The dataprocessing system 150 can compare the placement criteria with featuresof the web page, including the title, content, or metadata, anddetermine if the content item associated with the criteria should beprovided for display with the web page. For example, in someimplementations, the placement criteria may be the words “BrandNameProduct” and a placement server of the data processing system 150 isconfigured to place an associated content item on a page that includesboth “BrandName” and “Product.”

In some implementations, the placement system, e.g., data processingsystem 150, determines what content item to deliver with a particulardocument or web page based on a plurality of placement criteriaassociated with a plurality of content items. For example, the documentcan be parsed into a set of tokens. The tokens may be compared with theplacement criteria to identify associated content items. The placementcriteria may be expanded to include additional placement criteria. Theplacement system may determine the content item or items to deliverbased on original placement criteria or the expanded additionalplacement criteria.

In some implementations, the set of tokens parsed from the document maybe filtered to determine a sub-set of useful tokens for selectingcontent items. In some implementations, the set of tokens may beexpanded, in a manner similar to that described for placement criteria,to include additional tokens related to one or more of the tokens. Theplacement system, e.g., data processing system 150, may use the set oftokens, or the set of expanded tokens, in determining the content itemor items to deliver with the document.

FIG. 3 illustrates example relationships 300 between content items andplacement criteria. In some implementations, content 312 is associatedwith a criterion 322. Content may also be associated with multiplecriteria. For example, content 316 in the example of FIG. 3 isassociated with a first criterion 324 and a second criterion 326. Insome implementations, content may be grouped. For example, content 312may be grouped with similar or related content 316 into a content group310. The content group 310 may constitute a group of content items andcan exclude some content, e.g., unrelated content 395. The group 310 maybe associated with at least one placement criterion. The constituentcontent of the group 310 may be individually associated with respectivecriteria or, in some implementations, the content group 310 may beassociated with a group of placement criteria 320.

The placement criteria 320 can be expanded by the data processing system150 to include the criterion 322 and at least one expanded criterion332. In this example, the expanded criterion 332 may be associated withthe original criterion 322. In some implementations, the expandedcriterion 332 is associated (e.g., as represented by identifier 382)with the content 312. In this example, the placement criteria associatedwith the content 312 is expanded to include both the criterion 322(e.g., received at the data processing system 150 from the supplementalcontent provider 122) and the expanded criterion 332 (e.g., determinedby the data processing system 150).

The expanded criterion 322 can be added (as indicated by identifier 380)to the placement criteria 320 for the content 312 or for the contentgroup 310. In some implementations, a criterion is expanded to multiplecriteria, for example a criterion 324 can be expanded to a firstexpanded criterion 334 and a second expanded criterion 336. In someimplementations, multiple placement criteria are expanded into a singleexpanded criterion. For example, a first placement criterion 326 and asecond placement criterion 328 can be evaluated by the data processingsystem 150 to generate an expanded criterion 338. These are examples ofrelationships between content. The contents of content group 310,placement criteria 320, and criteria expansions 330, can include variousforms of content, content groups, or keywords or other placementcriteria having other relationships.

In some implementations, original placement criteria are expanded toinclude additional criteria when the original placement criteria aredetermined to be of sufficient quality with respect to their associatedcontent. For example, the data processing system 150 can determine thata placement criterion is successful if it leads to the placement ofcontent items with a web page where the viewer of the web page is likelyto respond to one of the content items. In some implementations, thedata processing system 150 can predict or estimate the value of aplacement criterion by determining at least one quality metric of theplacement criterion. For example, the quality metric module 158 candetermine a click through rate (or other performance trend) of aplacement criterion. In some implementations, placement criteria alreadyassociated with a content item are expanded to include a candidatecontent placement criterion when the quality metric for the placementcriteria already associated with the content item exceeds a thresholdsuch as a sufficiently high click through rate (CTR).

In one example of using a quality metric based on a click through rate(CTR), the CTR can be generated by evaluating the historical clickthrough rate of content items placed on web pages due to a correlationbetween a keyword or other placement criterion and the content of thoseweb pages. The content items used in generating a CTR may be selectedfrom a content group rather than all available content items. Forexample, the quality metric can be based on historical performance of aplacement criterion 322 (e.g., a keyword) relative to its correspondingcontent item 312 or content group 310, as opposed to a more genericquality metric of keyword performance relative to other content such asdifferent content items, e.g., content 316, or content items that arenot part of the content group such as unrelated content 395. The contentgroup may be based on similarity of product, content, sponsor, genre, orany other classification scheme. A content group may be formed throughan automated process, by a specialist, or even by specification from thecontent item's source. A quality metric based on CTR may also take intoaccount additional analysis based on statistical models to infer userbehavior. Quality metrics based on CTR generally estimate if a keywordis useful, for example, that a user is likely to click on a content itemplaced on a web page due to a correlation with that keyword. Otherquality metrics, such as a metrics based on click through volume, canalso be used to predict the value of candidate placement criteria suchas keywords.

With reference to FIGS. 1-3, in some implementations the data processingsystem 150 obtains at least one content placement criterion (e.g.placement criterion 322). The data processing system 150 can obtain thecontent placement criterion from the supplemental content provider 122or from the database 170, for example. The content placement criterioncan include a keyword or other placement criteria associated with acontent item, content group, or other content. The content placementmodule 154 can use the placement criteria to match its associatedcontent item with a web page, and the content item can be provided fordisplay with the web page at the user device 126.

In one implementation, the data processing system 150 determines aquality metric of the content placement criterion. For example, thequality metric module 158 can determine the click through rate of thecontent placement criterion relative to its associated content item. Thesame keyword may have different click through rates (or other qualitymetrics) for various content items. For example, a keyword may be highlyeffective for placing one content item, but the same keyword can beineffective when used to place a different content item. In oneimplementation, the data processing system 150 determines a qualitymetric for the content placement criterion relative to correspondingcontent.

For example, the data processing system 150 can obtain, e.g., from thesupplemental content provider 122, the keyword “tropical beachvacations” associated with a particular content item (or group ofcontent items) for an exclusive Caribbean resort. In this example, thedata processing system 150 can determine the click through rate of thekeyword “tropical beach vacations” relative to the particular contentitem for the Caribbean resort.

The data processing system 150 can also identify at least one candidatecontent placement criterion (e.g., criteria expansions 330). Thecandidate content placement criteria can be derived from the obtainedplacement criteria that are associated with the content item. Forexample, from the placement criterion “tropical beach vacations” thedata processing system 150 can identify the candidate placement criteria“beach vacations” “tropical beach” “beach holiday” “island getaway” or“honeymoon destination”. The candidate placement criteria generallyinclude keywords, terms, or other criteria that may be used to assistwith placement of the content item associated with the (e.g., original)placement criteria. The candidate placement criteria (e.g., “beachvacations”) can be terms of the original placement criteria (e.g.,“tropical beach vacations”) or different terms having a degree ofsemantic or subject matter similarity with the original placementcriteria (e.g., “beach holiday” since “holiday” is sometimes synonymouswith “vacation”).

In some implementations, the data processing system 150 expands criteriaassociated with the content to include the candidate content placementcriterion. For example, the data processing system 150 can evaluate thequality metric of the placement criterion associated with the contentitem, and can associate the candidate content placement criterion withthe content item when the quality metric of a placement criterionalready associated with the content item satisfies a threshold. Forexample, the candidate placement criterion can be added to placementcriteria of the content item when the click through rate of theplacement criterion relative to its associated content item is above athreshold level.

With reference to the above example, when the click through rate of thekeyword “tropical beach vacations” relative to the particular Caribbeanresort content item is above a threshold percentage, the data processingsystem 150 can expand placement criteria for the Caribbean resortcontent item to include at least one candidate keyword such as “beachholiday” or “island getaway” in addition to the criterion “tropicalbeach vacations”. In this example, the content placement module 154 canuse any of these criteria to place the Caribbean resort content item ona web page via the network 110 for display at the user device 126.

In some implementations, the data processing system 150 determines athrottling parameter of the content, and expands the placement criteriaof the content item based at least in part on the throttling parameter.The throttling parameter can be based on a monetary budget for contentitem placement. For example, an entity associated with the dataprocessing system 150 or the content publisher 120 may charge thesupplemental content provider 122 a fee to place content items on webpages. Excessive content placement may prematurely exhaust the budget.

The data processing system 150 can determine that the budget forplacement of a content item is insufficient, or is being depleted at asufficiently high rate that expanding the placement criteria to includeadditional criteria is not warranted. In this example, the dataprocessing system 150 can exclude a candidate content placementcriterion from association with content in order, for example, toconserve the budget dedicated to placement of that content item. In oneimplementation, the candidate placement criterion can be excludeddespite the quality metric of the (e.g., original) placement criteriahaving a high click through rate or otherwise satisfying a qualitymetric threshold. In some examples, the data processing system 150determines that a budget (or other throttling parameter) is sufficientand proceeds to associate candidate placement criteria with the contentitem.

FIG. 4 is a flow chart flow chart illustrating an example method 400 forexpanding placement criteria. In some implementations, the method 400obtains at least one content placement criterion associated with content(410) and determines a quality metric of the content placement criterion(430). In some implementations, the method 400 identifies a candidatecontent placement criterion (450), for example, by expanding theobtained criterion. In some implementations, the method 400 expands theplacement criteria associated with the content to include the contentplacement criterion and the candidate content placement criterion (480).

In some implementations, the method 400 expands the placement criteriaassociated with the content to include the candidate content placementcriterion (480) when the determined quality metric of the contentplacement criterion (430) indicates a valuable or successful placementcriterion. In some implementations, additional parameters areconsidered. For example, in some implementations, a throttling parametercan be used to limit criteria expansion.

Accordingly, in some implementations, determining a quality metric ofthe content placement criterion (430) includes an analysis of thecriterion and the objectives of the content item's source. One exampleof a quality metric is a click through rate (CTR), as discussed above.Other quality metrics, such as a click through volume, can also be usedto predict the value of candidate placement criteria such as keywords. Aquality metric may be a composite, e.g., a CTR adjusted by additionalmetric parameters. For example, the additional metric parameters may bea weight reflecting the frequency a candidate placement criterion isused in placing content items or the frequency a candidate placementcriterion occurs in documents with which content items may be delivered.

In some implementations, when the original placement criteria aresufficiently effective, the placement criteria specified by the contentprovider may be expanded. A criteria expansion system can generate a setof expanded criteria from one or more of the original placement criteriaspecified by the content provider. For example, an original criterionfor a content item highlighting a BrandName product may be the keyword“BrandName Product.” From this placement criteria, the expandedcriterion “BrandName” can be derived. This expanded criterion can beused in comparisons with web page content to place the content itemcorresponding to the “BrandName Product” criterion.

In some implementations, expansion of an original placement criterionmay be performed in an automated manner through phrase parsing (e.g.,using a subset of words found in the criterion or keyword), dictionarysubstitution (e.g., using synonyms), brand correlation substitution(e.g., using a table of model names for a BrandName product), orsemantic similarity, for example. In some implementations, criterionexpansion can be performed manually. For example, a specialist may haveadditional placement criterion suggestions not supplied by the contentitem's source. An expanded criterion may, but need not, include elementsfrom the original criterion, such as terms of a multi-term keyword.Expanded criteria can be stored in a repository (e.g., a database) andretrieved to assist with placement of content associated with theoriginal placement criterion.

In some situations, it may be useful to vary or restrict the use ofexpanded placement criteria. For example, the click through rate of acontent item might not be improved by expanding an ineffective originalplacement criterion, e.g., where the determined quality metric of theoriginal content placement criterion (430) is below some thresholdvalue. In one example, if the CTR of the original placement criterion islow, then the candidate placement criterion might not be used tosupplement the original criterion. (Because in this example, theoriginal keyword is not very good if it has a low CTR, so it isgenerally not beneficial to add a similarly ineffective expanded keywordto expand an ineffective original keyword). If the CTR is high, then thecandidate placement criterion can be used to supplement the originalplacement criterion for content item placement. The expanded placementcriterion can be used to complement the original placement criterionwhen the CTR of the original placement criterion for its correspondingcontent is sufficiently high. High and low CTR can be determined bycomparison with a threshold value, for example.

In another example, expanded placement criteria may be overlysuccessful. A content item provider (e.g., a sponsor) may wish tocontrol the number of presentations for a content item (e.g., using athrottling parameter) to maintain a preferred click through rate. Asponsor may want to limit the CTR for a number of reasons includingsponsorship budget, load balancing for a landing server, or otherlimitations (e.g., the content item includes an offer of a special dealfor a fixed number of participants). In these examples, the dataprocessing system can determine if expanded placement criteria should beadded to the original placement criteria.

For example, a sponsor can provide a maximum budget to be spent forcontent item placement, where the data processing system deducts acertain amount from that budget each time the corresponding content itemis placed on a web page. Placement criterion expansion may be restrictedby use of a throttling parameter to block candidate placement criteriaand conserve the budget, or when the budget is exhausted. For example,if a content item is running out of budget, then expansion can bereduced or otherwise restricted. This generally prevents the contentitem from being placed on a web page based on anything other than amatch with the original criterion provided by the sponsor. The expandedcriteria can be used to complement the original criteria when the budgetfor content item placement is sufficiently high, e.g., above a thresholdvalue.

FIG. 2, as introduced above, is a block diagram of a generalarchitecture of an illustrative computer system 200, as discussed above.The computer system or computing device 200 may be employed to implementany of the computer systems discussed herein, in accordance with someimplementations, including implementation of the system 100, contentpublisher 120, supplemental content provider 122, user device 126, dataprocessing system 150, content placement module 154, quality metricmodule 158, database (or repository) 170, and any other computer systemused. The computer system 200 can be used to provide information via thenetwork 110 and to evaluate data, e.g., to determine a quality metricsof a criterion, to expand placement criteria, and to use the expandedplacement criteria to identify content to provide for display with a webpage.

The computer system 200 of FIG. 2 can include one or more processors 250communicatively coupled to memory 270 and one or more communication ornetwork interfaces 210. Generally, the computer system 200 includes anI/O interface for receiving input via one or more input devices 224(e.g., a keyboard, mouse, or touch screen) and producing output to oneor more output devices 226 (e.g., a display, a screen, a touch screen,or speakers). The processors 250 can be included in the data processingsystem 150 or the other components of the system 100 such as the contentplacement module 154 or the quality metric module 158.

In the computer system 200 of FIG. 2, the memory 270 may comprise anycomputer-readable storage media, and may store computer instructionssuch as processor-executable instructions for implementing the variousfunctionalities described herein for respective systems, as well as anydata relating thereto, generated thereby, or received via thecommunications interface(s) or input device(s) (if present). Referringagain to the system 100 of FIG. 1, the content placement module 154, thequality metric module 158, or the database 170 can include the memory270 to store quality metrics, selection criteria, or content. Theprocessor(s) 250 shown in FIG. 2 may be used to execute instructionsstored in the memory 270 and, in so doing, also may read from or writeto the memory 270, and to cache 275, various information processed andor generated pursuant to execution of the instructions.

The processor 250 of the computer system 200 shown in FIG. 2 also may becommunicatively coupled to or control the communication or networkinterface(s) 210 to transmit or receive various information pursuant toexecution of instructions. For example, the network interface(s) 210 maybe coupled to a wired or wireless network, bus, or other communicationmeans and may therefore allow the computer system 200 to transmitinformation to and/or receive information from other devices (e.g.,other computer systems). While not shown explicitly in the system ofFIG. 1, one or more communications interfaces facilitate informationflow between the components of the system 100. In some implementations,the communications interface(s) may be configured (e.g., via varioushardware components or software components) to provide a website as anaccess portal to at least some aspects of the computer system 200.

The input device(s) 224 may be provided, for example, to allow a user tomake manual adjustments, make selections, enter data or otherinformation, or interact in any of a variety of manners with theprocessor during execution of the instructions. The output devices 226of the computer system 200 shown in FIG. 2 may be provided, for example,to allow information to be viewed or otherwise perceived in connectionwith execution of the instructions. The input device 224 and the outputdevice 226 may be the same device, such as a touch screen interface.

The processor 250 of the computer system 200 shown in FIG. 2 also may becommunicatively coupled to one peripheral interface(s) 230, e.g., a USBdevice, memory, disk drive, printer, or media device (e.g., a CD, DVD,or Blu-Ray player). Peripherals may be external to the computer system200, separated by an interface (e.g., a USB interface) or housed withinthe computer system 200 and more closely connected, (e.g., a SCSIhard-disk drive).

Implementations of the subject matter and the operations described inthis specification can be implemented in digital electronic circuitry,or in computer software, firmware, or hardware, including the structuresdisclosed in this specification and their structural equivalents, or incombinations of one or more of them. Implementations of the subjectmatter described in this specification can be implemented as one or morecomputer programs, i.e., one or more modules of computer programinstructions, encoded on computer storage medium for execution by, or tocontrol the operation of, data processing apparatus. The programinstructions can be encoded on an artificially generated propagatedsignal, e.g., a machine-generated electrical, optical, orelectromagnetic signal that is generated to encode information fortransmission to suitable receiver apparatus for execution by a dataprocessing apparatus. A computer storage medium can be, or be includedin, a computer-readable storage device, a computer-readable storagesubstrate, a random or serial access memory array or device, or acombination of one or more of them. Moreover, while a computer storagemedium is not a propagated signal, a computer storage medium can be asource or destination of computer program instructions encoded in anartificially generated propagated signal. The computer storage mediumcan also be, or be included in, one or more separate physical componentsor media (e.g., multiple CDs, disks, or other storage devices).

The features disclosed herein may be implemented on a smart televisionmodule (or connected television module, hybrid television module, etc.),which may include a processing circuit configured to integrate internetconnectivity with more traditional television programming sources (e.g.,received via cable, satellite, over-the-air, or other signals). Thesmart television module may be physically incorporated into a televisionset or may include a separate device such as a set-top box, Blu-ray orother digital media player, game console, hotel television system, andother companion device. A smart television module may be configured toallow viewers to search and find videos, movies, photos and othercontent on the web, on a local cable TV channel, on a satellite TVchannel, or stored on a local hard drive. A set-top box (STB) or set-topunit (STU) may include an information appliance device that may containa tuner and connect to a television set and an external source ofsignal, turning the signal into content which is then displayed on thetelevision screen or other display device. A smart television module maybe configured to provide a home screen or top level screen includingicons for a plurality of different applications, such as a web browserand a plurality of streaming media services (e.g., Netflix, Vudu, Hulu,etc.), a connected cable or satellite media source, other web“channels”, etc. The smart television module may further be configuredto provide an electronic programming guide to the user. A companionapplication to the smart television module may be operable on a mobilecomputing device to provide additional information about availableprograms to a user, to allow the user to control the smart televisionmodule, etc. In alternate implementations, the features may beimplemented on a laptop computer or other personal computer, asmartphone, other mobile phone, handheld computer, digital pad, tabletPC, or other computing device.

The operations described in this specification can be implemented asoperations performed by a data processing apparatus on data stored onone or more computer-readable storage devices or received from othersources. The term “data processing apparatus” or “computing device”encompasses all kinds of apparatus, devices, and machines for processingdata, including by way of example a programmable processor, a computer,a system on a chip, or multiple ones, or combinations, of the foregoing.The apparatus can include special purpose logic circuitry, e.g., an FPGA(field programmable gate array) or an ASIC (application specificintegrated circuit). The apparatus can also include, in addition tohardware, code that creates an execution environment for the computerprogram in question, e.g., code that constitutes processor firmware, aprotocol stack, a database management system, an operating system, across-platform runtime environment, a virtual machine, or a combinationof one or more of them. The apparatus and execution environment canrealize various different computing model infrastructures, such as webservices, distributed computing and grid computing infrastructures. Insome implementations, the content placement module 154 and the qualitymetric module 158 are special purpose logic circuits, e.g.,application-specific integrated circuits (ASICs). In someimplementations, the modules include one or more processors executinginstructions encoded as firmware or software on communicatively coupledcomputer readable media. In some implementations, the modules caninclude or share one or more data processing apparatuses, computingdevices, or processors.

A computer program (also known as a program, software, softwareapplication, script, or code) can be written in any form of programminglanguage, including compiled or interpreted languages, declarative orprocedural languages, and it can be deployed in any form, including as astand alone program or as a module, component, subroutine, object, orother unit suitable for use in a computing environment. A computerprogram may, but need not, correspond to a file in a file system. Aprogram can be stored in a portion of a file that holds other programsor data (e.g., one or more scripts stored in a markup languagedocument), in a single file dedicated to the program in question, or inmultiple coordinated files (e.g., files that store one or more modules,sub programs, or portions of code). A computer program can be deployedto be executed on one computer or on multiple computers that are locatedat one site or distributed across multiple sites and interconnected by acommunication network.

The processes and logic flows described in this specification can beperformed by one or more programmable processors executing one or morecomputer programs to perform actions by operating on input data andgenerating output. The processes and logic flows can also be performedby, and apparatuses can also be implemented as, special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application specific integrated circuit).

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any kind of digital computer. Generally, aprocessor will receive instructions and data from a read only memory ora random access memory or both. The essential elements of a computer area processor for performing actions in accordance with instructions andone or more memory devices for storing instructions and data. Generally,a computer will also include, or be operatively coupled to receive datafrom or transfer data to, or both, one or more mass storage devices forstoring data, e.g., magnetic, magneto optical disks, or optical disks.However, a computer need not have such devices. Moreover, a computer canbe embedded in another device, e.g., a mobile telephone, a personaldigital assistant (PDA), a mobile audio or video player, a game console,a Global Positioning System (GPS) receiver, or a portable storage device(e.g., a universal serial bus (USB) flash drive), for example. Devicessuitable for storing computer program instructions and data include allforms of non volatile memory, media and memory devices, including by wayof example semiconductor memory devices, e.g., EPROM, EEPROM, and flashmemory devices; magnetic disks, e.g., internal hard disks or removabledisks; magneto optical disks; and CD ROM and DVD-ROM disks. Theprocessor and the memory can be supplemented by, or incorporated in,special purpose logic circuitry.

To provide for interaction with a user, implementations of the subjectmatter described in this specification can be implemented on a computerhaving a display device, e.g., a LCD (liquid crystal display) monitor,LED or OLED screen, a CRT (cathode ray tube), a plasma screen, or aprojector, for displaying information to the user and a touch screen,keyboard, or a pointing device, e.g., a mouse or a trackball, by whichthe user can provide input to the computer. Other kinds of devices canbe used to provide for interaction with a user as well; for example,feedback provided to the user can be any form of sensory feedback, e.g.,visual feedback, auditory feedback, or tactile feedback; and input fromthe user can be received in any form, including acoustic, speech, ortactile input. In addition, a computer can interact with a user bysending documents to and receiving documents from a device that is usedby the user; for example, by sending web pages to a web browser on auser's client device in response to requests received from the webbrowser.

The subject matter described in this specification can be implemented ina computing system that includes a back end component, e.g., as a dataserver, or that includes a middleware component, e.g., an applicationserver, or that includes a front end component, e.g., a client computerhaving a graphical user interface or a Web browser through which a usercan interact with an implementation of the subject matter described inthis specification, or any combination of one or more such back end,middleware, or front end components. The components of the system can beinterconnected by any form or medium of digital data communication,e.g., a communication network. Examples of communication networksinclude a local area network (“LAN”) and a wide area network (“WAN”), aninter-network (e.g., the Internet), and peer-to-peer networks (e.g., adhoc peer-to-peer networks).

The computing system such as system 200 or system 100 can includeclients and servers. A client and server are generally remote from eachother and typically interact through a communication network. Therelationship of client and server arises by virtue of computer programsrunning on the respective computers and having a client-serverrelationship to each other. In some implementations, a server transmitsdata (e.g., an HTML page) to a client device (e.g., for purposes ofdisplaying data to and receiving user input from a user interacting withthe client device). Data generated at the client device (e.g., a resultof the user interaction) can be received from the client device at theserver.

In some implementations, the systems and methods of the disclosure canrestrict keyword expansion for a content item based on a determined CTRof the original keyword for that content item. In some implementations,the systems and methods of the disclosure can restrict keyword expansionfor a content item based on a budget allotted to the content item. Insome implementations, placement criteria other than a keyword may beused and expanded as shown.

While this specification contains many specific implementation details,these should not be construed as limitations on the scope of anyinventions or of what may be claimed, but rather as descriptions offeatures specific to particular implementations of particularinventions. Certain features described in this specification in thecontext of separate implementations can also be implemented incombination in a single implementation. Conversely, various featuresdescribed in the context of a single implementation can also beimplemented in multiple implementations separately or in any suitablesub-combination. Moreover, although features may be described above asacting in certain combinations and even initially claimed as such, oneor more features from a claimed combination can in some cases be excisedfrom the combination, and the claimed combination may be directed to asub-combination or variation of a sub-combination. Similarly, whileoperations are depicted in the drawings in a particular order, thisshould not be understood as requiring that such operations be performedin the particular order shown or in sequential order, or that allillustrated operations be performed, to achieve desirable results. Incertain circumstances, multitasking and parallel processing may beadvantageous. Moreover, the separation of various system components inthe implementations described above should not be understood asrequiring such separation in all implementations, and it should beunderstood that the described program components and systems cangenerally be integrated in a single software product or packaged intomultiple software products.

Where technical features in the drawings, detailed description or anyclaim are followed by reference signs, the reference signs have beenincluded for the sole purpose of increasing the intelligibility of thedrawings, detailed description, and claims. Accordingly, neither thereference signs nor their absence have any limiting effect on the scopeof any claim elements. References to “or” may be construed as inclusiveso that any terms described using “or” may indicate any of a single,more than one, and all of the described terms.

Thus, particular implementations of the subject matter have beendescribed. Other implementations are within the scope of the followingclaims. In some cases, the actions recited in the claims can beperformed in a different order and still achieve desirable results. Inaddition, the processes depicted in the accompanying figures do notnecessarily require the particular order shown, or sequential order, toachieve desirable results. In certain implementations, multitasking andparallel processing may be advantageous.

What is claimed is:
 1. A computer-implemented method of evaluatinginformation in a computer network environment, comprising: obtaining, bya computing system, initial distribution criteria used to place content;identifying, by the computing system, a first candidate distributioncriterion and a second candidate distribution criterion; identifying afrequency at which the first candidate distribution criterion is used inplacing first content items; determining that the frequency at which thefirst candidate distribution criterion is used in placing the firstcontent items exceeds a threshold; evaluating one or more factors todetermine that a set of distribution criteria used to place the contentis to be expanded; and in response to (1) determining that the set ofdistribution criteria used to place the content is to be expanded, and(2) determining that the frequency at which the first candidatedistribution criterion is used in placing the first content itemsexceeds the threshold, expanding, by the computing system, the set ofdistribution criteria used to place the content to include the initialdistribution criteria and the first candidate distribution criterion. 2.The method of claim 1, further comprising: identifying a frequency atwhich the second candidate distribution criterion is used in placingsecond content items; determining that the frequency at which the secondcandidate distribution criterion is used in placing the second contentitems does not exceed the threshold; and excluding the second candidatedistribution criterion from the set of distribution criteria used toplace the content in response to determining that the frequency at whichthe second candidate distribution criterion is used in placing thesecond content items does not exceed the threshold.
 3. The method ofclaim 1, further comprising: determining that the set of distributioncriteria used to place the content is to be further expanded;identifying a frequency at which the second candidate distributioncriterion occurs within a plurality of documents with which contentitems may be delivered; determining that the frequency at which thesecond candidate distribution criterion occurs within the plurality ofdocuments with which content items may be delivered exceeds a secondthreshold; and in response to (1) determining that the set ofdistribution criteria used to place the content is to be furtherexpanded, and (2) determining that the frequency at which the secondcandidate distribution criterion occurs within the plurality ofdocuments with which content items may be delivered exceeds the secondthreshold, expanding, by the computing system, the set of distributioncriteria used to place the content to include the second candidatedistribution criterion.
 4. The method of claim 3 wherein the pluralityof documents with which content items may be delivered comprises aplurality of webpages.
 5. The method of claim 1, further comprising:determining the first candidate distribution criterion from at least oneterm drawn from the initial distribution criteria.
 6. The method ofclaim 1, wherein the content includes a media element, the methodfurther comprising: using the first candidate distribution criterion toidentify a correlation between the content and a web page; and providingthe media element for display with the web page responsive to thecorrelation.
 7. The method of claim 1, wherein the content is from acontent group having a plurality of group content distribution criteriaincluding the initial distribution criteria, the method furthercomprising: determining a quality metric for the content based on aclick through rate of at least one of the group content distributioncriteria.
 8. The method of claim 7, wherein a plurality of content itemsin the content group are included in the content group based onsimilarity of one or more of product, content, sponsor, or genre.
 9. Themethod of claim 1, further comprising: obtaining a third candidatedistribution criterion associated with the content; determining that abudget for placing the content satisfies a budget threshold; and addingthe third candidate distribution criterion to the set of distributioncriteria used to place the content in response to determining that thebudget for placing the content satisfies the budget threshold.
 10. Themethod of claim 1, further comprising: obtaining a third candidatedistribution criterion associated with the content; determining that abudget for placing the content fails to satisfy a budget threshold; andexcluding the third candidate distribution criterion from the set ofdistribution criteria used to place the content in response todetermining that the budget for placing the content fails to satisfy thebudget threshold.
 11. A system for providing information via a computernetwork, comprising: a data processing system having a content placementmodule on one or more processors; a memory storing instructions that,when executed by the one or more processors, cause the data processingsystem to perform operations comprising: obtaining initial distributioncriteria used to place content; identifying a first candidatedistribution criterion and a second candidate distribution criterion;identifying a frequency at which the first candidate distributioncriterion is used in placing first content items; determining that thefrequency at which the first candidate distribution criterion is used inplacing the first content items exceeds a threshold; evaluating one ormore factors to determine that a set of distribution criteria used toplace the content is to be expanded; and in response to (1) determiningthat the set of distribution criteria used to place the content is to beexpanded, and (2) determining that the frequency at which the firstcandidate distribution criterion is used in placing the first contentitem exceeds the threshold, expanding the set of distribution criteriaused to place the content to include the initial distribution criteriaand the first candidate distribution criterion.
 12. The system of claim11, the operations further comprising: identifying a frequency at whichthe second candidate distribution criterion is used in placing secondcontent items; determining that the frequency at which the secondcandidate distribution criterion is used in placing the second contentitems does not exceed the threshold; and excluding the second candidatedistribution criterion from the set of distribution criteria used toplace the content in response to determining that the frequency at whichthe second candidate distribution criterion is used in placing thesecond content items does not exceed the threshold.
 13. The system ofclaim 11, the operations further comprising: determining that the set ofdistribution criteria used to place the content is to be furtherexpanded; identifying a frequency at which the second candidatedistribution criterion occurs within a plurality of documents with whichcontent items may be delivered; determining that the frequency at whichthe second candidate distribution criterion occurs within the pluralityof documents with which content items may be delivered exceeds a secondthreshold; and in response to (1) determining that the set ofdistribution criteria used to place the content is to be furtherexpanded, and (2) determining that the frequency at which the secondcandidate distribution criterion occurs within the plurality ofdocuments with which content items may be delivered exceeds the secondthreshold, expanding, by the computing system, the set of distributioncriteria used to place the content to include the second candidatedistribution criterion.
 14. The system of claim 13 wherein the pluralityof documents with which content items may be delivered comprises aplurality of webpages.
 15. The system of claim 11, the operationsfurther comprising: determining the first candidate distributioncriterion from at least one term drawn from the initial distributioncriteria.
 16. The system of claim 11, wherein the content includes amedia element, the operations further comprising: using the firstcandidate distribution criterion to identify a correlation between thecontent and a web page; and providing the media element for display withthe web page responsive to the correlation.
 17. The system of claim 11,wherein the content is from a content group having a plurality of groupcontent distribution criteria including the initial distributioncriteria, the operations further comprising: determining a qualitymetric for the content based on a click through rate of at least one ofthe group content distribution criteria.
 18. The system of claim 17,wherein a plurality of content items in the content group are includedin the content group based on similarity of one or more of product,content, sponsor, or genre.
 19. The system of claim 11, the operationsfurther comprising: obtaining a third candidate distribution criterionassociated with the content; determining that a budget for placing thecontent satisfies a budget threshold; and adding the third candidatedistribution criterion to the set of distribution criteria used to placethe content in response to determining that the budget for placing thecontent satisfies the budget threshold.
 20. A computer readable storagemedium having instructions stored thereon to provide information on acomputer network, the instructions comprising instructions that, whenexecuted by one or more computers, cause the one or more computers toperform operations comprising: obtain initial distribution criteria usedto place content; identify a first candidate distribution criterion anda second candidate distribution criterion; identify a frequency at whichthe first candidate distribution criterion is used in placing firstcontent items; determine that the frequency at which the first candidatedistribution criterion is used in placing the first content itemsexceeds a threshold; evaluate one or more factors to determine that aset of distribution criteria used to place the content is to beexpanded; and in response to (1) determining that the set ofdistribution criteria used to place the content is to be expanded, and(2) determining that the frequency at which the first candidatedistribution criterion is used in placing the first content itemsexceeds a threshold, expand the set of distribution criteria used toplace the content to include the initial distribution criteria and thefirst candidate distribution criterion.